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Why Automate? 



• Every 2 Weeks 

• Multiple Branches 

• ~3000 Clock Hours 
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Why Automate? 


New 

Existing 


Launch 


Cost of Testing Content 
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Why Automate? 


Live Game NA/EU 


Story Up 
Hotfixes 




China Launch 


MTX 

New Player 
Experience 

Certification 
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Feature Packs 


Gameplay 

Updates 

Balance 

Changes 
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Democratizing Results 
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Enabling Testability Overview 

• Available Tools / Resources 

• Motivations for Change 

• Testability Requirements 

• Functional Interaction 

• Frameworks 



Game Client 
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Available Tools / Resources 


LUA Script 
Console 

Instrumentation 


r 1 

Bot Commander 

Local 1 

Network 1 

1 AWS 

Clients 1 

1 Clients 
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Motivations for Change 

• Performance Impact 

• Flexibility 

• Core module support 

• Marshalling 

• Knowledge 

• Platform and Game Independence 

• Multiple Platforms 
. Multiple Titles 
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Testability Requirements 

• Client and Character Isolation 

• No Graphics Cards 

• Content Streaming 

• Client Distribution 

• Instrumentation 
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Functional Interaction 


Game Client I Test Framework 


Gameplay 
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Functional Interaction 
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Handlers 


Game Client 



Handlers 
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Observers 

• Time sensitive observability 

• Limited Scope 
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Relays 



• Direct Access to Server APIs 

• Bypass UI Interactions 

• Test is the Driver 
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Frameworks 


Game Client 


Handlers 


Observers 


Relay 
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r ^ 

Test Framework 

L A 
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Test 
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Frameworks 


Data Access Layer 


Connector 



Socket 


Protocol 


JSON 


Python 

Magic 
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Frameworks 
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/Client/Player/Move 
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Frameworks 
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player = game. player 

position = player.position 

direction = position + Vector3. FORWARD 

movement = player.move(direction) 
movement. wait() 

assert player.position != position 
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Summary 

• Instrumentation based on test scenarios 

• Focus on business value 

• Build what you need 

• Be mindful of your dependencies 

• Think about tools as well as tests 
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Supporting Parallelism Overview 

• Why Parallel Execution? 

• Defining Testable Chunks 

• Modeling Concurrent Testing 

• Refining the Model 

• Metrics 
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Why Parallel Execution? 




■ New 

■ Existing 
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Why Parallel Execution? 



Core 
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Why Parallel Execution? 


Execution Time/Test 


Automated 


Manual 



■ Setup 

■ Execution 

■ Cleanup 


0 
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10 
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Defining Testable Chunks 


Profession 


Warrior 


Signets 

Banners 

Shouts 


Mesmer 


{ 


Mantras 

Clones 


Thief 


Venoms 

Tricks 


• Isolated Content 

• Same Suite of Tests 

• Many Variants 


Traps 
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Defining Testable Chunks 


Profession 


Warrior 


Signets 

Banners 

Shouts 


Mesmer 


{ 


Mantras 

Clones 


Thief 


Venoms 

Tricks 


• Organization 

• Specialized Metadata 

• Inheritance 

• Constraints 


Traps 
























GAME DEVELOPERS CONFERENCE® 2015 


MARCH 2-6, 2015 GDCONF.COM 


Modeling Concurrent Testing 


• Fragile Interactions 

• Unreliable Messaging 

• Complex Services 

• Multiple Contracts 











Modeling Concurrent Testing 

• Stateless Services 

• Processing Queues 

• Single Responsibility 

• Transformative 
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Modeling Concurrent Testing 

• Aggregation 

• Transforms 

• Reclamation 

• Provisioning 

• Results 
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Refining the Model 



ntdecS) 




CouchDB 

relax 
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simple, flexible, fun 


TypeScript 


^ redis 




GAME DEVELOPERS CONFERENCE® 2015 


Metrics 

• Test Throughput 

• Test Pass Cadences 

• Resource Utilization 

• Queue Duration 

• Completion Time 


MARCH 2-6, 2015 GDCONF.COM 


2 

seconds / test 

5+ 

runs / day 

40 

clients / server 

1 

min / test 

50 

min / run 
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Summary 

• Use Test Markup To Define Constraints 

• Model Execution Requirements 

• Avoid "Not Invented Here" Solutions 

• Eliminate Bottlenecks 

• Keep It Simple 
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Democratizing Results Overview 

• Customers of Test Automation 

• Reaching Your Audience 

• Different Perspectives 

• Keeping It Simple 
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Customers of Test Automation 


Release Manaqement 


Product 

Health 

Integration 

Readiness 


Programmin 


Areas of 
Failure 

Severity 



Content 

Integration 

Assumptions 


QA Embeds 


Branch 

Health 



Regressions 
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Reaching Your Audience(s) 

• Interdependent Customers 

• Unique Goals 


Actionable Information 
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Different Perspectives 


Static 

Analysis 


Integration 

Testing 


Crash 

Reporting 
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Keeping It Simple 



Product | gw2 | Branch | gw2-dev|102) I | Reporting - ) | Scheduling"! 


Filter | Build fl | 

Priority [T] [ 7 ) [|] 0 Status | All | 


Time Prtortty Branch Build » Status Duration | 20 ^ _ I 


Time TestGUID 

Log Status Duration | 2 ^ ; | 



7A 


Verification name 

Status Reason Lorem ipsum dolor sit amet, consectetur adipisicing elit 

Verification name 

Status Reason Lorem ipsum dolor sit amet, consectetur adipisicing elit 

Verification name 

Status Reason Lorem ipsum dolor sit amet, consectetur adipisicing elit 

Verification name 

Status Reason Lorem ipsum dolor sit amet, consectetur adipisicing elit 


Verification name Status Reason lorem ipsum dolor sit a met, consectetur adipisicing elit 


Product [GuiidWars2 | Branch pjvingworidi ~| Reporting Test Schedule Test Templates 


Guild Wars 2 > LivingWor1d2 > Build 32562 > A Guardian Skill 

Verification: Should do all sorts of things 
An actual failure reason 


Download Log 




Filter 

Priority 

HI 

0 12 3 


Guild Wars 2 LivingWorldl 

Build 32567 Priority 0 


Guild Wars 2 
Build 32562 

LKringWorldl 
Priority 0, 1 

6/24/148:15 


GuidWars2 

LivingWorldl 

6/23/14 18:20 

Build 32535 

Priority 1, 2, 3 



GuidWars2 

LivingWorldl 

6/23/14 17:52 

Build 32535 

Priority 0 



GuldWars2 

LivingWorldl 

6/23/1417:52 

Build 32535 

Priority 0 
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Keeping It Simple 


Branch [i 


Reporting Scheduling 


DEV 


Build Tim* 

420/0 11/1//2014 16:22 
42070 11/17/2014 16:21 

Status 


Status 


EMI 

EMI 


Complete 
Complete 

Nam* 

Iron Marches Map Should Load 
Successfully 

Whisper Should Forward Messages 
Between Different Maps 
Whisper Should Forward Messages 
Between Mayers 
Alpine Iceburg Story Oungeon 
Should Be Enter able At Expected 
Level 

Armarsmith Bronze Ingot Recipe 
Should Not Give Items When There 
Are No Ingredients 
Diessa Plateau Map Should Load 
Successfully 

Dragon Repeat Map Should Load 
Successfully 

Huntsman Bronze Ingot Recipe 
Should Give Correct Rewards On 
Use 

Karma Boosts Should Increase 
Karma Gained By 50% From Event 
Leatherworker Bolt Of Jute Recipe 
Should Consume ktgredienls On Use 
Match ors Leap Map Should Load 
Successfully 

PUyer Should Be Able To Use 
Huitsman Discipline At Level 
Mayer Should Be Able To Use Tailor 
Discipline At Level 
Risen Dragon Story Dungeon Should 
Be Enterahle At Expected Level 


Unable To Transition To Map: 322/aadc-81f5-4e62-811a-C36b3c49366f 
Message Was Not Received 
Message Was Not Received 


Timeout 300s 


[42:0:0:189] 
[42:0:0:189] 
Timeout 300s 
[42:0:0:189) 
[42:0:0:189] 
[42:0:0:189] 


| Scripts 

k analytics 
k buildmon 
k buildsetver 

k gw2 
k misc 

| OPTIONS 

■ Notifications 
- Display All 
BulldGw2 Builds 

SCjjEPULER 
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CHANGE BRANCH 


■ 

# Time 

Status 

Branch Script Owner 

Parameters 

13558 12/8/14 18:06 

Complete 

Dev 

gw2. servers 


13557 12/8/14 18:02 

Complete 

Dev 

gw2.game 


13556 12/8/1417:22 

Complete 

Dev 

gw2. audio, buildaudio 


13555 12/8/14 17:09 

Complete 

Dev 

gw2.audio.integratevoice — — 


13554 12/8/14 17:05 

Complete 

Dev 

gw2 . voi celoc. u pdateloc 


13553 12/8/14 17:04 

Complete 

Dev 

gw2.game 


13552 12/8/14 16:22 

Complete 

Dev 

gw2. audio, buildaudio 


13551 12/8/1416:13 

Complete 

Dev 

gw2.audio.integratevoice — 


13550 12/8/14 16:08 

Complete 

Dev 

gw2.game 


13549 12/8/14 15:26 

Complete 

Dev 

gw2. audio, buildaudio * 


13548 12/8/14 15:23 

Complete 

Dev 

gw2.game 


13547 12/8/14 14:41 

Complete 

Dev 

gw2.game 


13546 12/8/14 13:45 

Complete 

Dev 

gw2. audio, buildaudio 


13545 12/8/14 13:17 

Complete 

Dev 

gw2 . commerce . data 


13543 12/8/14 12:55 

Complete 

Dev 

gw2.assets.composi te 


13541 12/8/14 12:54 

Complete 

Dev 

gw2.game 


13542 12/8/14 12:33 

Canceled 

dev 

gw2 . commerce .data 


13544 12/8/14 12:27 

Canceled 

Dev 

gw2.assets.composi te 


13540 12/8/14 12:12 

Complete 

Dev 

gw2.game 


13539 12/8/1411:27 

Canceled 

Dev 

gw2.game 


13538 12/8/14 11:19 

Complete 

Dev 

gw2.audio.integratevoice 


13537 12/8/14 10:35 

Complete 

Dev 

gw2. voi celoc. updateloc * - 
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Development 510 Available Tests 1 


Build 

Time 

Status 

Failures 

Blocked 

+ 44575 

01/30/2015 17:54 

99% 

1 

287 

+ 44575 

01/30/2015 15:51 

Complete 

2 

159 

44575 

01/30/2015 15:23 

Complete 

0 

0 

| 44575 

01/30/2015 14:06 

Complete 

0 

0 

+ 44575 

01/30/2015 12:03 

Complete 

1 

0 

44575 

01/30/2015 11:41 

Cancelled 

0 

0 

44575 

01/30/2015 11:39 

Cancelled 

0 

0 

+ 44575 

01/30/2015 11:39 

Cancelled 

0 

84 

+ 44575 

01/30/2015 11:30 

Complete 

2 

36 

+ 44575 

01/30/2015 10:05 

Complete 

0 

232 

+ 44575 

01/30/2015 08:03 

Complete 

0 

343 

+ 44575 

01/30/2015 06:01 

Complete 

5 

332 

+ 44575 

01/30/2015 03:59 

Complete 

0 

33 

+ 44575 

01/30/2015 01:57 

Complete 

0 

338 

+ 44575 

01/29/2015 23:57 

Complete 

0 

1 

44575 

01/29/2015 21:55 

Complete 

0 

0 

+ 44564 

01/29/2015 19:41 

Complete 

1 

204 

+ 44564 

01/29/2015 18:57 

Complete 

3 

1 

+ 44564 

01/29/2015 18:41 

Complete 

2 

0 

+ 44564 

01/29/2015 18:08 

Complete 

2 

1 

+ 44540 

01/29/2015 12:39 

Complete 

2 

0 
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Summary 

• Know What Your Customers Want 

• Reports Need To Be Actionable 

• Focus on Interesting Results 

• Augment Existing Customer Workflows 

• Build Trustworthy Experiences 



% 0 
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Questions? 


@lancehit 

lance@arena.net 
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